Skip to content

Add JSON to Parquet Variant conversion with shredding - #8391

Merged
AdamGS merged 9 commits into
developfrom
adamg/variant-json-handling
Jun 23, 2026
Merged

Add JSON to Parquet Variant conversion with shredding#8391
AdamGS merged 9 commits into
developfrom
adamg/variant-json-handling

Conversation

@AdamGS

@AdamGS AdamGS commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR adds a vortex.json_to_variant expression for converting UTF-8 or vortex.json extension arrays into Variant arrays.

The expression lives in vortex-json so JSON-facing expression metadata and serialization are independent of a concrete Variant encoding. vortex-parquet-variant provides the concrete execution kernels, parsing JSON through parquet_variant_compute and producing Parquet Variant storage.

@AdamGS AdamGS mentioned this pull request Jun 12, 2026
@AdamGS
AdamGS force-pushed the adamg/variant-json-handling branch from 860196c to b9ae199 Compare June 12, 2026 16:27
@codspeed-hq

codspeed-hq Bot commented Jun 12, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 2 improved benchmarks
❌ 1 regressed benchmark
✅ 1580 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation slice_empty_vortex 310 ns 368.3 ns -15.84%
Simulation rebuild_naive 111 µs 98.9 µs +12.32%
Simulation bitwise_not_vortex_buffer_mut[128] 273.6 ns 244.4 ns +11.93%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing adamg/variant-json-handling (8318ee1) with develop (71319b7)

Open in CodSpeed

@AdamGS
AdamGS force-pushed the adamg/variant-json-handling branch 3 times, most recently from 3e823bc to 0bc0405 Compare June 17, 2026 11:30
@AdamGS AdamGS added the changelog/feature A new feature label Jun 17, 2026
@AdamGS
AdamGS force-pushed the adamg/variant-json-handling branch 3 times, most recently from 67592eb to 6b3a236 Compare June 17, 2026 15:43
@AdamGS
AdamGS marked this pull request as ready for review June 17, 2026 15:59
@AdamGS
AdamGS requested a review from a team June 17, 2026 15:59
@AdamGS
AdamGS force-pushed the adamg/variant-json-handling branch from 6b3a236 to d78dcaa Compare June 17, 2026 15:59
@AdamGS AdamGS changed the title Json-Variant exprs Add parquet-variant transformation to/from JSON, including shredding Jun 17, 2026
@AdamGS
AdamGS requested review from mprammer and robert3005 June 17, 2026 15:59
gatesn
gatesn previously requested changes Jun 18, 2026

@gatesn gatesn left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One sec!

Comment thread encodings/parquet-variant/src/fns/json_to_variant.rs Outdated
@gatesn

gatesn commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Do these functions mix DType (JSON) with encoding (ParquetVariant)? At least the docs seem to suggest that they do.

We probably want vortex-json to expose to_json and from_json functions (or maybe this just a cast kernel) and then implement them using execute_parent for ParquetVariant.

But it seems like you are actually defining a more general DType::JSON -> DType::Variant (any variant encoding) function? In which case, it cannot be a cast because you want the shredding options, but the definition should probably still live in the vortex-json crate, and implement general-purpose shredding (per the canonical Variant array), without touching the non-shredded data?

Comment thread encodings/parquet-variant/src/fns/variant_to_json.rs Outdated
Comment thread encodings/parquet-variant/src/fns/variant_to_json.rs Outdated
Comment thread encodings/parquet-variant/src/fns/json_to_variant.rs Outdated
@AdamGS

AdamGS commented Jun 18, 2026

Copy link
Copy Markdown
Contributor Author

@gatesn I've really tried to keep expressions that can't be evaluated directly from the core. I thought execute_parent rules were less flexible but I guess I can try and do that.

@AdamGS
AdamGS force-pushed the adamg/variant-json-handling branch from a13a0a2 to 8b284a0 Compare June 23, 2026 10:02
AdamGS added 5 commits June 23, 2026 11:02
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
@AdamGS
AdamGS force-pushed the adamg/variant-json-handling branch from 8b284a0 to d6355b9 Compare June 23, 2026 10:02
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
@AdamGS
AdamGS force-pushed the adamg/variant-json-handling branch from d6355b9 to 8cb09a9 Compare June 23, 2026 10:04
@AdamGS AdamGS changed the title Add parquet-variant transformation to/from JSON, including shredding Add JSON to Parquet Variant conversion with shredding Jun 23, 2026
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
@AdamGS
AdamGS requested review from connortsui20 and gatesn June 23, 2026 10:24
@AdamGS

AdamGS commented Jun 23, 2026

Copy link
Copy Markdown
Contributor Author

I think I addressed the comments. I've also made the PR a bit smaller and more focused, just on json_to_variant which I think is the more interesting code path.

@robert3005 robert3005 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the only thing is the implicit JsonToVariant for VarBinView. We should require an extension json type at this level.

Comment thread encodings/parquet-variant/src/kernel.rs Outdated
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
@AdamGS
AdamGS requested a review from robert3005 June 23, 2026 15:06
@AdamGS
AdamGS dismissed gatesn’s stale review June 23, 2026 15:27

addressed

@AdamGS
AdamGS merged commit 682c7b8 into develop Jun 23, 2026
78 of 79 checks passed
@AdamGS
AdamGS deleted the adamg/variant-json-handling branch June 23, 2026 15:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants